The present invention relates to an image processing method and apparatus for processing an image obtained using, e.g., an image sensing device, and an image sensing apparatus.
In recent years, image sensing devices such as CCD image sensing devices are popularly used as image reading elements for television cameras, video cameras, electronic cameras, and the like. However, the image sensing device is inferior to silver halide media in terms of the dynamic range, and an image obtained by an image sensing device has considerably low quality depending on the photographing conditions.
In order to expand the dynamic range of an image sensing device, a technique for sensing an image of an identical scene in different exposure amounts to obtain a plurality of image signals, and combining the plurality of image data by a predetermined calculation to obtain an image with an expanded dynamic range is proposed in, e.g., Japanese Patent Laid-Open No. 63-306778.
FIGS. 1A to 1C and FIGS. 2A to 2C are graphs showing typical conventional methods of expanding the dynamic range.
The graph in FIG. 1A shows the relationship between an illuminance (abscissa) and an output from the image sensing device (ordinate) obtained upon sensing a certain object. In FIG. 1A, a characteristic I represents the relationship between an illuminance and an output from the device for an image obtained upon sensing the object in a proper exposure (to be referred to as a "standard image" hereinafter). A characteristic II represents the relationship between an illuminance and an output from the device for an image obtained upon sensing the object in an over exposure (to be referred to as a "non-standard image" hereinafter). Referring to FIG. 1A, signals below an output level n are regarded as noise. A characteristic I' is obtained when an output value from the image sensing device (Y-axis) of the standard image I is multiplied by a luminance level adjustment value K.
A concept of image combining according to this method will be described. In FIG. 1A, at a position "x.sub.1 " on the axis representing the object illuminance, as for the image signal (image signal I) obtained in a proper exposure, the output from the image sensing device is at noise level. However, the image signal (image signal II) obtained in an over exposure has a proper value (exceeds the noise level n). That is, when an output value between level 0 and the noise level n is replaced with the output value of the non-standard image II, an image signal with an expanded dynamic range can be obtained. As shown in FIG. 1A, the gradient of the standard image I obtained in a proper exposure is different from that of the non-standard image obtained in an over exposure. Therefore, when the signal value I is only replaced with the signal value II, a discontinuous region is generated. For replacement, not the image signal I but an image signal I' is used.
In FIG. 1A, the image signal I' (to be also referred to as a standard image hereinafter) is obtained by multiplying the standard image I by the predetermined luminance level adjustment value K. The level adjustment value K is set such that the gradient of the standard image I' coincides with that of the non-standard image signal II. Processing of generating the standard image I' will be referred to as "luminance level adjustment" hereinafter. With this luminance level adjustment, a noise level n' of the standard image I' becomes n.times.K.
An appropriate threshold value T is set above the noise level n' (=n.times.K). As shown in FIG. 1B, data in a region below the threshold value T is replaced with the data of the non-standard image II. As for a region above the threshold value T, the data of the standard image I' is used without being processed. With this processing, noise in the section (n'-n) of the standard image I' is removed. The gradient of a combined image III (FIG. 1B) obtained in this manner corresponds to the gradient of the standard image I multiplied by K. Therefore, as shown in FIG. 1C, when the gradient of the combined image III is multiplied by 1/K to obtain the original gradient, the luminance level corresponding to the standard image I can be obtained. With the above processing, combined image data III' with an expanded dynamic range, i.e., at a low noise level can be obtained.
The conventional technique for expanding the dynamic range by decreasing noise regions has been described above. A technique for expanding the dynamic range by decreasing saturated regions will be described below with reference to FIGS. 2A to 2C.
FIGS. 2A to 2C are graphs for explaining a technique for combining an image sensed in a proper exposure (to be referred to as a standard image Iv hereinafter) with an image sensed in an under exposure (to be referred to as a standard image V hereinafter).
When image data is represented as 8-bit data, and the output level exceeds 1,023, the image data is saturated. Referring to FIG. 2A, at "x2" on the axis representing the object illuminance, the image data of the standard image IV is saturated while the image data of the standard image V has a proper value. Therefore, as shown in FIG. 2A, the gradient of a non-standard image V', which is obtained by multiplying the gradient of the standard image V by the luminance level adjustment value K', i.e., by multiplying the data of the standard image V by K', coincides with that of the standard image IV. As a result, the non-standard image V' obtains a luminance level corresponding to that of the standard image IV. As in noise removal (FIG. 1A), an appropriate threshold value T' is set. The image data of the standard image IV in a region above the threshold value T' is replaced with the non-standard image V'. As for a region below the threshold value T', the image data of the standard image IV is used without being processed. With this processing, the data of a combined image VI (FIG. 2B) has the same luminance level (gradient) as that of the standard image IV in FIG. 2A, though it has a 10-bit width. Hence, an appropriate knee point is set, as shown in FIG. 2C, and a bright region is compressed while maintaining the same gradient as that of the standard image IV.
With the above processing, image data VI' with an expanded dynamic range without saturation in a bright portion can be combined.
In the above conventional techniques, a dark noise region or a saturated region of a standard image is determined using a predetermined threshold value (T or T'). In the conventional techniques, when the image sensing device has a color filter, an output signal from the image sensing device is used without being processed to set a threshold value.
However, output values from image sensing devices vary in accordance with colors because of the spectral sensitivity, so the dark noise or saturated region cannot be completely determined. This problem will be described below referring to a case wherein a saturated region of a standard image sensed in a proper exposure is determined and replaced with a corresponding region of a non-standard region.
When the image sensing device has three color filters for R, G, and B, saturated regions in each of R, G, and B image data may not be determined and separated unless the characteristics of each color filter is taken into consideration. For this reason, saturated regions undesirably remain in combined image data. In addition, it becomes difficult to set a threshold value because of variations in each color image signal.